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1.  SUMMARY 

1.1.  The  Vision  Machine 

Much  of  our  work  during  the  past  year  has  focused  on  building  our  Vision 
M^w;hine  system.  The  Vision  Machine  is  a  testbed  for  our  research  on  parallel 
vision  algorithms  and  their  integration.  The  system  consists  of  an  input  device 
-  a  movable  two  camera  Eye-Head  system  with  six  degrees  of  freedom  -  and 
the  16K  Connection  Machine  (CM-1).  We  have  concentrated  on  implementing 
and  testing  early  vision  algorithms,  and  on  developing  a  new  sophisticated 
technique  for  their  integration.  The  output  of  the  integration  stage  will  be 
used  for  navigation  and  recognition  tasks. 

1.2.  An  Overview 

From  August  31,  1986  to  August  31,  1987,  we  have  been  using  the  Connec¬ 
tion  Machine  delivered  on  July  31,  1986  by  Thinking  Machines  Corporation 
(TMC).  We  have  developed  and  tested  a  substantial  body  of  vision  software 
on  the  machine.  We  have  also  nearly  completed,  well  ahead  of  schedule,  the 
development  of  an  integrated  Vision  Machine  that  includes  several  early  vi¬ 
sion  algorithms,  and  the  integration  stage  of  middle  vision.  As  outlined  in 
our  original  proposal,  we  have  begun  to  explore  parallel  algorithms  at  the 
higher  level  of  recognition.  We  have  also  studied  the  p>erformance  of  alterna¬ 
tive,  nonconventional  architectures  for  navigation,  and  worked  on  the  difficult 
issue  of  alternative  parallel  languages  for  the  Connection  Machine,  in  addition 
to  *LISP  and  C*.  The  body  of  this  report  gives  an  overview  of  the  results 
of  our  research  during  the  second  twelve  months  of  funding.  Details  cein  be 
found  in  the  attached  publications. 
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2.  ACHIEVEMENTS  IN  THE  SECOND  YEAR 


We  give  below  a  brief  overview  of  our  main  achievements. 

2.1.  The  Vision  Machine 

The  Vision  Machine  hardware  consists  of  a  “Eye-Head”  system  and  a  Con¬ 
nection  Machine  (CM-1).  The  MIT  Eye-Head  system  is  a  movable  robot  that 
consists  of  two  CCD  cameras,  two  sets  of  two  mirrors  each,  and  a  “head” 
that  can  be  moved  with  two  degrees  of  freedom  by  step-motors.  Both  the 
“head”  and  the  mirrors  (driven  by  galvanometers)  are  computer  controlled. 
The  system  can  effectively  look  around  the  ninth  floor  of  the  MIT  Artificial 
Intelligence  Laboratory  building  at  Technology  Square,  and  grab  stereo  im¬ 
ages.  In  the  very  near  future,  we  will  add  zoom,  auto-focusing  and  color 
capabilities  to  this  input  device. 

The  Connection  Machine  [Hillis,  1985]  is  a  fine-grained  parallel  computing 
machine  having  up  to  64K  processors,  and  operating  under  a  single  instruc¬ 
tion  stream  broadcast  to  all  processors.  The  MIT  Connection  Machine  is  a 
16K  processor  CM-1.  Each  processor  is  a  1-bit  serial  processor,  with  4K  bits 
of  memory.  There  are  two  modes  of  communication  among  the  processors.  In 
the  first  mode,  the  processors  can  be  thought  of  as  connected  by  a  mesh  of 
wires  into  an  m  x  m  grid  network  (the  NEWS  network,  so-named  for  the  four 
cardinal  directions),  allowing  rapid  direct  communication  between  neighbor¬ 
ing  processors;  m  is  variable.  The  second  mode,  the  router,  allows  messages 
to  be  sent  from  amy  processor  to  any  other  processor  in  the  machine.  The 
processors  in  the  Connection  Machine  can  be  envisioned  as  the  vertices  of 
a  16-dimensional  hypercube.  Each  processor  in  the  Connection  Machine  is 
identified  by  a  unique  integer,  its  hypercube  address.  Messages  pass  along 
the  edges  of  the  hypercube  from  source  processors  to  destination  processors. 
Besides  local  operations  in  the  processors,  the  Connection  Machine  can  return 
to  the  host  machine  the  result  of  various  operations  on  a  field  in  all  processors; 
it  can  return  the  global  maximum,  minimum,  sum,  logical  AND,  and  logical 
OR  of  the  field. 

The  core  of  the  Vision  Machine  system  is  software  on  the  CM-1  writ¬ 
ten  mainly  in  *LISP  and  PARIS,  which  are  additions  to  Common-Lisp  that 
provide  access  to  the  Connection  Machine.  Since  the  Connection  Machine  is 
a  Single-Instruction  Multiple-Data  (SIMD)  machine,  there  is  an  instruction 
stream  broadcast  to  the  Connection  Machine  from  the  host,  in  our  case  a  Sym¬ 
bolics  Lisp  Machine.  The  form  of  programming  suitable  for  the  Connection 
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Machine,  due  to  its  SIMD  nature,  is  “data- level  parallelism”,  which  focuses 
on  the  movement  smd  interaction  of  data.  Each  processor  in  the  Connection 
Machine  is  associated  with  a  data  element.  All  selected  data  elements  in  a 
SIMD  machine  are  manipulated  in  the  same  fashion.  Because  of  these  fea¬ 
tures  of  the  Connection  Machine,  it  is  natural  to  map  pixels  in  the  images  we 
analyze  to  processors,  the  data  elements  in  the  Connection  Machine. 

The  Connection  Machine  architecture  provides  some  useful  primitive  op¬ 
erations  for  early,  middle  and  high-level  vision.  It  is  distinguished  from  mesh 
computers  by  its  capability  for  general  non-local  communication  among  the 
processors,  and  from  pyramid  machines  by  its  uniform  bandwidth  communi¬ 
cation  among  all  processors. 

Our  integration  effort  has  to  this  point  concentrated  on  integrating  data 
produced  by  individual  modiiles  into  a  coherent  description  of  the  scene, 
specifically  at  boimdaries.  The  memory  limitations  of  our  CM-1  force  us 
to  run  each  module  serially,  since  it  is  not  possible  to  represent  simultaine- 
ously  in  the  CM  1  all  of  the  many  structures  involved  in  the  texture,  color, 
motion  and  stereo  modules.  With  a  CM-2,  we  should  be  able  to  couple  these 
modules  more  tightly. 

2.2.  Early  Vision  Algorithms 


2.2.1.  Edge  Detection 

Edge  detection  is  an  important  first  step  in  many  low-level  vision  algorithms. 
It  generates  a  concise,  compact  description  of  the  structure  of  the  image, 
suitable  for  manipxilation  in  higher-level  interpretation  tasks.  We  have  imple¬ 
mented  on  the  Connection  Machine  both  the  Marr-Hildreth  and  the  Canny 
edge  detection  schemes. 

A  ftmdamentsd  operation  in  vision  processing  is  filtering  the  input  image 
to  remove  noise  and  to  select  an  appropriate  spatial  sceile.  Typically,  filtering 
is  accomplished  by  convolution  with  a  filter  of  boimded  spatial  extent,  often 
a  Gaussian.  We  have  implemented  a  variety  of  methods  for  computing  the 
Gaussian  convolution  of  an  image.  Primarily,  we  use  the  binomial  approxi¬ 
mation  to  the  one-dimensional  Gaussian  implemented  by  repeated  summing, 
using  a  kernel  of  1/4, 1/2, 1/4.  To  smooth  the  image  with  a  filter  approximat¬ 
ing  a  Gaussian  with  standard  deviation  a  requires  2<7^  —  1  convolutions  with 
this  kernel.  Each  convolution  needs  two  additions  and  two  NEWS  accesses. 
This  method  can  also  be  used  to  generate  an  approximation  to  convolution 
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with  a  two-dimensional  Gaussian.  Convolution  with  a  Gaussian  filter  can 
also  be  approximated  by  iterated  convolution  with  a  uniform  (boxcar)  filter 
of  width  N  and  height  1/N.  To  approximate  a  Gaussian  with  standard  devia¬ 
tion  <7,  12<r^  —  1/N^  —  1  iterations  are  required.  This  approximation  is  useful 
when  <T  is  Isurge.  We  compute  *  G((t)  by  convolving  with  G{(t),  retaining 
extra  precision,  and  then  filtering  with  the  discrete  Laplacian. 

To  detect  zero-crossings  in  the  Maxr-Hildreth  method,  each  processor 
need  only  examine  the  sign  bits  of  neighboring  processors,  using  NEWS  ac¬ 
cesses. 

Canny  Edge  Detection 

The  Canny  edge  detector  is  based  on  directional  derivatives,  so  it  has  im¬ 
proved  localization.  Implementing  the  Canny  edge  detector  on  the  Connec¬ 
tion  Machine  involves  implementing  Gaussiein  filtering,  computing  directional 
derivatives,  non-maximum  suppression,  and  thresholding  with  hysteresis.  The 
algorithm  has  been  implemented  on  the  Connection  Machine,  and  is  routinely 
used  in  real-time  processing  tasks  as  an  integral  part  of  the  Vision  Machine 
system. 

Gaussian  filtering  and  computing  directional  derivatives  are  local  opera¬ 
tions  as  described  above.  In  non-maximum  suppression,  gradient  magnitudes 
are  interpolated  along  lines  in  gradient  directions.  Then  those  pixels  are  se¬ 
lected  for  which  the  gradient  magnitude  is  a  local  maximum.  Thresholding 
with  hysteresis  estimates  the  distribution  of  gradient  magnitudes  in  the  image, 
and  chooses  a  low  threshold,  low,  below  which  all  points  are  considered  noise. 
Then  a  high  threshold,  high,  is  computed;  pixels  with  gradient  magnitudes 
above  high  are  automatically  marked  as  edge  elements.  Any  pixels  above  low 
threshold  but  below  high  become  edge  elements  only  if  they  can  be  connected 
to  a  selected  pixel  above  high  by  a  set  of  selected  pixels  (local  maxima),  above 
low.  This  requires  propagating  information  along  curves. 

Histograms 

Estimating  the  gradient  magnitude  distribution  is  performed  by  computing 
its  histogram.  There  are  several  ways  this  can  be  implemented  on  the  Con¬ 
nection  Machine.  Gradient  magnitudes  can  be  quantized  for  the  histogram 
bucket  size.  Sorting  these  values  reconfigures  the  data:  ...k,k,k,k,k  +  l,k  + 
1,  A:  -f  1  —  Each  processor  determines  whether  its  left  neighbor  is  less  than 
itself.  Each  processor  for  which  this  holds  sends  its  cube  address  to  location 
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Hk  in  the  histogram  table,  resulting  in  a  cumulative  frequency  distribution 
table,  easily  converted  to  a  histogram.  Computing  a  histogram  in  m  buckets 
by  counting  involves  stepping  from  0  <  i  <  m,  selecting  processors  where 
intensity  =  t,  and  counting  the  number  of  selected  processors  (Hi),  using  m 
global  counting  operations.  When  m  is  less  than  64,  this  can  be  more  effi¬ 
cient  than  sorting.  Finally,  when  only  one  value  in  the  distribution  is  needed, 
for  example,  finding  the  fc**  percentile,  the  percentile  can  be  found  in  a  bi¬ 
nary  search,  using  O(log  m)  global  counting  operations.  Computing  a  Hough 
Transform  is  similar  to  computing  a  histogram.  In  this,  we  will  be  able  to 
take  advantage  of  a  priori  information  on  the  distribution  of  values  to  devise 
a  fast  algorithm. 


Propagation 


In  thresholding  with  hysteresis,  the  existence  of  a  high  value  on  a  curve  is 
propagated  along  the  curve,  to  enable  any  low  pixels  to  become  edge  pixels. 
Only  pixels  above  low  which  survive  non-maximum  suppression  are  consid¬ 
ered.  Each  pixel  can,  in  constant  time,  find  the  neighboring  pixels  with  which 
it  forms  a  connected  line,  using  the  NEWS  network.  All  pixels  above  high  are 
marked  as  edge  pixels.  Currently,  the  program  iterates,  in  each  step  marking 
as  edge  pixels  any  low  pixels  adjacent  to  edge  pixels,  using  NEWS  connections. 
When  no  pixels  change  state,  the  iteration  terminates,  taking  a  number  of 
steps  proportionsd  to  the  length  m  of  the  longest  chain  of  low  pixels  which 
eventually  become  edge  pixels.  Using  doubling,  propagating  the  edge  property 
changes  this  to  O(logm).  In  practice,  propagation  in  the  NEWS  network  is 
faster  than  using  the  asymptotically  optimal  doubling  procedure. 


2.2.2.  Stereo 


Binocular  stereo  is  one  of  the  most  precise  sources  of  depth  information.  For 
several  months  we  have  been  using  a  parallel  implementation  of  a  new  stereo 
algorithm  on  the  Connection  machine.  This  stereo  algorithm  is  related  to 
schemes  proposed  by  Marroquin  [1983j,  Prazdny  [1985],  and  especially  Marr 
and  Poggio  [1976].  These  algorithms  are  feature  based.  We  have  sdso  recently 
explored  an  algorithm  based  on  matching  intensities  directly.  It  is  conceivable 
that  a  similM  scheme  may  complement  a  feature  based  algorithms  to  provide 
a  denser  depth  map. 
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Parallel  Stereo  and  the  Forbidden  Zone  Constraint 

A  new  simple  but  fast  algorithm  has  been  implemented  in  collaboration  with 
Thinking  Machines  Corporation  on  the  Connection  Machine  [Drumheller  and 
Poggio,  1986].  In  the  past  months  it  has  been  reimplemented  and  improved 
by  W.  Gillett  as  an  integral  part  of  the  Vision  Machine  system.  Its  features 
include:  a)  the  potential  for  combining  different  primitives,  including  color 
information;  b)  the  use  of  a  new  and  stronger  formulation  of  the  uniqueness 
constraint;  and  c)  a  dispairity  representation  that  maps  efficiently  into  the  CM 
architectiue.  The  algorithm  consists  of  the  following  steps: 

Compute  features  for  matching 

Compute  potential  matches 

Determine  the  amount  of  local  support  for  each  potential  match 

Choose  correct  matches  on  the  basis  of  local  support  and  constraints  on  unique¬ 
ness  and  ordering. 

The  algorithm  does  not  require  a  particular  type  of  matching  feature. 
Different  types  of  features  can  be  used,  such  as  the  sign  of  the  convolution 
with  a  difference  of  Gaussians.  Let  us  assume  that  the  images  are  p>erfectly 
registered  and  that  all  epipolar  lines  are  horizontal.  We  can  represent  the  set 
of  potential  matches  in  a  one -dimensional  stereo  matching  problem  (i.e.,  for  a 
pair  of  epipolar  lines)  by  the  same  diagram  iised  by  Marr  and  Poggio  [1976]. 
This  representation  of  the  stereo  problem  is  the  starting  point  for  mapping  a 
stereo  algorithm  into  the  CM. 

Potential  matches  are  allowed  to  occur  between  two  zero-crossings  of  the 
same  sign.  This  implements  the  compatibility  constraint  [Marr  and  Poggio, 
1976].  The  first  step  toward  distinguishing  the  correct  matches  from  the  false 
ones  is  to  apply  the  continuity  constraint  [Marr  and  Poggio,  1976].  This  prin¬ 
ciple  states  that  since  most  surfaces  in  the  real  world  are  piecewise  smooth, 
potential  matches  should  be  selected  that  result  in  a  piecewise  smooth  dis¬ 
parity  function.  A  straightforward  way  to  measure  how  well  each  disparity 
satisfies  the  smoothness  condition  is  to  convolve  the  three-dimensional  region 
of  i-y-d— space  contained  by  the  field  P  with  a  three-dimensional  kernel  that 
gathers  support  from  smooth  configurations  of  potential  matches.  There  are 
many  different  kernels,  or  support  functions,  which  will  do  a  good  job  on  this 
task.  Marr  and  Poggio  [1976]  use  a  very  simple  support  function  (or  “excita¬ 
tory  region”)  that  is  circiilar,  uniformly  weighted  and  flat,  i.e.,  it  occupies  only 
one  level  in  the  disparity  dimension.  Every  potential  match  is  surrounded  by 
an  hourglass- shaped  forbidden  zone.  Within  the  forbidden  zone  there  must  be 
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no  more  than  one  match  iinless  the  scene  contains  transparent  or  narrowly- 
occluding  objects.  Examples  of  such  special  scenes  include  a  pane  of  glziss  with 
markings  on  both  sides,  or  a  vertical  wire  suspended  in  front  of  a  textured  wall. 
These  situations  violate  the  ordering  constraint  [Yuille  and  Poggio,  1984].  If 
we  assume  that  the  scene  contains  only  opaque  objects  with  no  narrow  occlu¬ 
sions,  then  it  makes  sense  to  enforce  uniqueness  not  only  along  lines  of  sight, 
but  along  any  line  of  sight  in  the  forbidden  zone.  A  correct  match  should  be 
the  only  match  within  its  entire  forbidden  zone.  The  Marr-Poggio  algorithm 
and  the  wiimer-take-all  algorithms  mentioned  earlier  [Pra^zdny,  1985;  Pollard, 
Mayhew  and  Frisby,  1985;  Marroquin,  1983]  use  only  the  left  and  right  eye 
lines  of  sight,  which  comprise  a  small  subregion  of  the  entire  forbidden  zone. 
Notice  that  the  forbidden  zone  property  is  reflexive:  if  a  match  lies  within  the 
forbidden  zone  of  another,  the  latter  is  in  the  forbidden  zone  of  the  first  [see 
Yuille  and  Poggio,  1985,  in  which  trzinsitivity  is  also  proven]. 

The  algorithm  enforces  imiqueness  by  suppressing  8dl  matches  that  lie 
within  the  forbidden  zone  of  the  match  that  gathers  mfiximum  support  from 
the  3-D  convolution  operation.  The  process  of  non-maximum  suppression 
along  lines  of  sight  is  called  the  winner-take-all  approach.  We  have  used 
a  stronger,  more  general  version  of  the  winner-take-all  method.  Instead  of 
applying  non-maximum  suppression  along  only  the  left  and  right  eye  lines  of 
sight,  we  apply  it  across  the  entire  forbidden  zone.  In  general,  the  use  of  the 
entire  forbidden  zone  in  the  winner-take-all  step  results  in  fewer  matches  than 
when  using  just  the  left  and  right  eye  lines  of  sight.  However,  the  number  of 
errors  almost  always  decreases  more  than  the  number  of  matches,  especially  in 
the  occluded  region.  Therefore,  the  ratio  of  errors  to  matches  decreeises.  This 
supports  the  hypothesis  that  the  entire  forbidden  zone  could  be  exploited  to 
advantage  for  scenes  known  to  contain  only  opaque  objects  with  no  narrow 
occlusions. 

W.  Gillett  has  recently  implemented  a  very  simple  statistical  analysis  of 
the  voting  process  (how  close  the  winners  axe,  and  where  ties  occur)  to  obtmn 
some  initial  information  about  depth  discontinuities  during  stereo  matching, 
rather  than  afterwards  as  in  the  MRF  scheme.  Preliminary  experiments  indi¬ 
cate  that  the  method  is  feasible. 

The  stereo  algorithm  runs  on  the  Connection  Machine  system  with  good 
results  on  natural  scenes  in  times  that  are  typically  on  the  order  of  one  second. 
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2.2.3.  Motion 


A  Parallel  Constraint  Algorithm  for  Optical  Flow 

J.  Little,  H.  BiilthofF  and  T.  Poggio  have  developed  a  new,  fast  parallel  algo¬ 
rithm  for  computing  optical  flow.  The  algorithm  is  based  on  a  new  regular¬ 
ization  method  that  we  call  the  “constraint  method”.  This  method,  based  on 
a  theorem  of  Tikhonov,  can  enforce  local  constraints  and  lead  directly  to  effi¬ 
cient,  parallel  algorithms.  The  specific  constraint  exploited  by  our  algorithm 
can  be  shown  to  correspond,  in  its  most  general  form,  to  3-D  rigid  motion  of 
planar  surfaces.  Segmentation  of  the  motion  field  czin  be  obtained  from  the 
optical  flow  field  generated  by  the  algorithm.  An  iterative  scheme  provides 
fast  approximate  solutions  and  subsequently  refines  them.  The  algorithm  has 
been  implemented  on  the  CM,  and  is  routinely  used  in  real-time  processing 
tasks  as  an  integral  part  of  the  Vision  Machine  system. 

The  algorithm  generates  an  optical  flow  field,  a  vector  field  which  approx¬ 
imates  the  projected  motion  field.  The  procedure  produces  sparse  or  dense 
output,  depending  on  whether  edge  features  or  intensities  are  used.  The  algo¬ 
rithm  assumes  that  image  displacements  are  small,  within  a  range  (±^,±(5). 
In  addition,  it  is  assumed  that  the  optical  flow  is  locally  constant  in  a  small 
region  sxirroimding  a  point.  This  assumption  is  strictly  only  true  for  transla¬ 
tional  motion  of  3-D  planar  surface  patches  parallel  to  the  image  plane.  It  is 
a  restrictive  assumption  which,  however,  may  be  a  satisfactory  local  approxi¬ 
mation  in  many  cases.  Let  Et{x,y)  and  y)  represent  transformations 

of  two  discrete  images  separated  by  time  interval  At,  such  as  filtered  images 
or  a  map  of  the  intensity  changes  in  the  two  images  (more  generally,  they  can 
be  maps  containing  a  feature  vector  at  each  location  (x,  y)  in  the  image). 

We  look  for  a  discrete  motion  displacement  v  =  {vx,Vj,)  at  each  location 
X,  y  in  the  image  such  that 


ll-E^iCa^iy)  -  +  UiA<,y  -f-i;yA<)||p*tchi  =  (^) 

where  the  norm  is  a  summation  over  a  local  neighborhood  centered  at  each  lo¬ 
cation  (x,  y);  v(x,  y)  is  assumed  to  be  constant  in  the  neighborhood.  Equation 
(1)  implies  that  we  should  look  at  each  (x,y)  for  v  =  (vr,  Vj,)  such  that 


I  {&t{x,y)  -  Et+£it{x  +  Vi  At,  y  -I-  Vy  A<))^dxdy 

patch) 


(2) 


is  minimized.  Alternatively,  one  can  maximize  the  negative  of  the  integrated 
result.  Equation  (2)  represents  the  sum  of  the  pointwise  squared  differences 
between  a  patch  in  the  first  image  centered  around  the  location  (x,y)  and  a 
patch  in  the  second  image  centered  around  the  location  (i  +  Vj. At,  y  +  VyAt). 

This  algorithm  can  be  translated  easily  into  the  following  description. 
Consider  a  network  of  processors  representing  the  result  of  the  integrand  in 
equation  (2).  Assume  for  simplicity  that  this  result  is  either  0  or  1  (this  is  the 
case  if  Et  and  Et+^t  are  binary  feature  maps).  The  processors  hold  the  result 
of  differencing  (taking  the  logical  “exclusive  or”)  of  the  right  and  left  image 
map  for  different  values  of  (x,y)  and  Vx,Vy.  The  next  stage,  corresponding 
exactly  to  the  integral  operation  over  the  patch,  is  for  each  processor  to  sum- 
mate  the  total  (2)  in  an  (x,y)  neighborhood  at  the  same  disparity.  Each 
processor  thus  collects  a  vote  indicating  support  for  a  patch  of  surface  ex¬ 
isting  at  that  displacement.  The  algorithm  iterates  over  all  displacements  in 
the  range  ±^,  ±^,  recording  the  values  of  the  integral  (2)  for  each  displace¬ 
ment.  The  last  stage  is  to  choose  v(x,y)  from  among  the  displacements  in 
the  allowed  range  that  maximizes  the  integral.  This  is  done  by  an  operation 
of  “non-maximum  suppression”  across  velocities  from  the  finite  allowed  set: 
at  the  given  (x,  y),  the  processor  is  foimd  that  has  the  maximum  vote.  The 
corresponding  y.(x,y)  is  the  velocity  of  the  surface  patch  found  by  the  algo¬ 
rithm.  The  actual  implementation  of  this  scheme  can  be  simplified  so  that 
the  “non-maximum  suppression”  occurs  during  iteration  over  displacements, 
so  that  no  actual  table  of  summed  differences  over  displacements  need  be 
constructed.  In  practice,  this  formalism  has  been  shown  to  be  effective  both 
for  synthetic  and  natural  images  using  many  image  transformations  before 
comparison,  including  edge  detectors  (both  zero-crossings  of  the  Laplacian  of 
Gaussian  and  Canny’s  method)  which  generate  speuse  results  along  intensity 
edges,  and  direct  use  of  intensities  or  the  sign  of  the  Laplacian  of  Gaussian, 
which  generate  dense  results. 

2.2.4.  Texture 

Texture  computations,  specifically,  determining  texture  boundairies,  is  an  im¬ 
portant  component  of  the  Vision  Machine.  Such  boundaries  provide  yet 
another  cue  to  determine  object  botmdaries,  since  the  projection  of  object 
boimdaries  often  coincides  with  texture  changes.  Voorhees  [1987]  has  de¬ 
scribed  several  schemes  for  reliably  determining  texture  boundaries  in  images. 
One  simple  scheme  first  performs  filtering  with  the  Laplacian  of  Gaussian, 
then  thresholds  the  filter  image  at  some  non-zero  value,  yielding  blobs  which 
characterize  the  textural  elements  in  the  textured  regions.  To  compute  the 
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density  of  the  blobs,  or  the  amount  of  area  locally  covered  by  the  blobs,  it  is 
sufficient  to  sum  locally  the  number  of  pixels  which  are  in  the  blobs.  This  gives 
a  useful  discriminator  which  can  identify  some  textures.  This  algorithm  h8is  a 
particularly  simple  parallel  implementation  on  the  Connection  Machine,  since 
there  exist  routines,  relying  on  the  scan  capability  of  the  Connection  Machine, 
which  can  rapidly  compute  such  local  sums  everywhere  in  an  image.  These 
computations  are  prohibitively  expensive  on  a  serial  computer.  Further  en¬ 
hancements  of  these  texture  discriminators  involve  orientation  selectivity  and 
thinning  operations  on  the  blobs  derived  above.  We  are  actively  working  on 
these  developments:  they  also  have  simple  implementations  on  the  Connec¬ 
tion  Machine.  Other  textmal  discrimination  methods  summate,  for  example, 
the  energy  in  the  image  filtered  by  the  Laplacian  of  Gaussian.  This  fits  into 
the  framework,  and  appears  to  be  useful  in  some  cases. 


2.3.  The  MRF  Integration  Scheme 


Integration  of  several  vision  modules  is  likely  to  be  one  of  the  main  keys  to 
the  power  and  robustness  of  the  human  visual  system.  The  problem  of  inte¬ 
grating  early  vision  cues  is  clearly  the  central  problem  in  our  Vision  Machine 
project,  E.  Gamble  and  T.  Poggio  have  suggested  that  integration  is  best 
performed  at  the  location  of  discontinuities  in  early  processes,  such  as  dis¬ 
continuities  in  depth,  intensity,  motion  and  texture.  They  have  used  coupled 
Markov  Random  Field  (MRF)  models  based  on  Bayesian  estimation  tech¬ 
niques,  to  combine  vision  modalities  with  their  discontinuities.  They  have 
derived  a  scheme  to  integrate  intensity  edges  with  stereo  depth  and  motion 
field  information,  and  show  results  on  synthetic  and  natural  images.  The 
use  of  intensity  edges  to  integrate  other  visual  cues  and  to  help  to  discover 
discontinuities  emerges  as  a  general  and  powerful  principle.  These  models 
generate  algorithms  that  map  naturally  on  parallel  fine-grained  eurchitectures 
such  as  the  Connection  Machine.  Gamble  and  Poggio  have  chosen  to  use  the 
machinery  of  Markov  Random  Fields,  initially  suggested  for  image  processing 
by  Geman  and  Geman  [1984].  Consider  the  prototypical  problem  of  approx¬ 
imating  a  sxirface  given  sparse  and  noisy  data  (depth  data)  on  a  regular  2-D 
lattice  of  sites.  We  first  define  the  prior  probability  of  the  class  of  surfaces  we 
are  interested  in.  The  probability  of  a  certain  depth  at  any  given  site  in  the 
lattice  depends  only  upon  neighboring  sites  (the  Markov  property).  Because 
of  the  Clifford- Hammersley  theorem,  the  prior  probability  is  guaranteed  to 
have  the  Gibbs  form: 
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P(/)=;|e-^  (3) 

where  Z  is  a  normalization  constant,  T  is  called  temperature,  and  U{f)  = 
Ylc  ^cif)  is  an  energy  function  that  can  be  computed  zis  the  sum  of  local 
contributions  from  each  neighborhood.  The  sum  of  the  potentials,  Uc(,X),  is 
over  the  neighborhood’s  cliques.  A  clique  is  either  a  single  lattice  site  or  a 
set  of  lattice  sites  such  that  any  two  sites  belonging  to  the  set  axe  neighbors 
of  one  another.  Thus  U(f)  csui  be  considered  as  the  sum  over  the  possible 
configurations  of  each  neighborhood  [see  Marroquin  et.al.,  1987;  Gamble  and 
Poggio,  1987].  As  a  simple  example,  when  the  surfaces  axe  expected  to  be 
smooth,  the  prior  probability  can  be  given  in  terms  of: 

Ucif)  =  ifi  -  fi?  (4) 

where  i  and  j  are  neighboring  sites  (belonging  to  the  same  clique). 

If  a  model  of  the  observation  process  is  available  (i.e.,  a  model  of  the 
noise),  then  one  can  write  the  conditional  probability  P{gff)  of  the  sparse 
observation  g  for  any  given  surface  /.  Bayes  Theorem  then  allows  one  to  write 
the  posterior  distribution: 


PU/9)  =  (5) 

In  the  simple  earlier  example,  we  have  (for  Gaussian  noise): 

nf/s)  =  -  g>f  +  (/,  - 

c 

where  7i  =  1  only  where  data  are  available.  More  complicated  cases  can  be 
handled  in  a  similar  manner  [Gamble  and  Poggio,  1987]. 

The  posterior  distribution  cannot  be  solved  analytically,  but  sample  dis¬ 
tributions  with  the  probability  distribution  of  Equation  (5)  can  be  obtained 
using  Monte  Carlo  techniques  such  as  the  Metropolis  algorithm.  These  algo¬ 
rithms  sample  the  space  of  possible  surfaces  according  to  the  probability  dis¬ 
tribution  P(f/g),  which  is  determined  by  the  prior  knowledge  of  the  allowed 
class  of  surfaces,  the  model  of  noise,  and  the  observed  data.  In  our  imple¬ 
mentation,  a  highly  parallel  computer  generates  a  sequence  of  surfaces  from 
which,  for  instance,  the  surface  corresponding  to  the  maximum  of  P{f/g)  can 
be  found.  This  corresponds  to  finding  the  global  minimum  of  U{f/g)  (simu¬ 
lated  annealing  is  one  of  the  possible  techniques).  Other  criteria  can  be  used: 
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Maxroqmn  [1985]  has  shown  that  the  average  surface  /  under  the  posterior 
distribution  is  often  a  better  estimate  and  can  be  obtained  more  efficiently  by 
simply  finding  the  average  value  of  /  at  each  lattice  site. 

One  of  the  main  attractions  of  MRF  is  that  the  prior  probability  dis¬ 
tribution  can  be  made  to  embed  more  sophisticated  assumptions  about  the 
world.  Geman  and  Geman  [1984]  introduced  the  idea  of  another  process,  the 
line  process,  located  on  the  dual  lattice  and  representing  explicitly  the  pres¬ 
ence  or  absence  of  discontinuities  that  break  the  smoothness  assumption.  The 
associated  energy  then  becomes: 


V  -  c(/)  =  (/,  -  fi\l  -  l{)  +  /3Vc(i;)  (6) 

where  /  is  a  binary  line  element  between  site  Vc  is  a  term  that  retleccs  the 
fact  that  certain  configurations  of  the  line  process  are  more  likely  than  others 
to  occur.  In  our  world,  depth  discontinuities  are  usually  themselves  continu¬ 
ous  and  non-intersecting,  and  are  rarely  isolated  joints.  These  properties  of 
physical  discontinuities  can  be  enforced  locally  by  defining  an  appropriate  set 
of  energy  values  V’c(l)  for  different  configurations  of  the  line  process  in  the 
neighborhood  of  the  site. 

It  is  obviously  possible  to  extend  the  energy  function  to  accommodate 
the  interaction  of  more  processes  and  their  discontinuities.  In  particuleir, 
Geunble  and  Poggio  have  extended  the  energy  function  to  couple  several  of 
the  early  vision  modules  (depth,  motion,  texture  and  color)  to  intensity  edges 
in  the  image.  This  is  a  central  point  in  our  integration  scheme:  intensity 
edges  guide  the  computation  of  discontinuities  in  the  other  physical  processes, 
thereby  coupling  sxirface  depth,  surface  orientation,  motion,  texture  and  color 
each  to  the  image  intensity  data  and  thus  to  each  other.  The  reason  for  the 
role  of  intensity  edges  is  clear:  changes  in  surface  properties  usually  produce 
large  intensity  gradients  in  the  image.  It  is  exactly  for  this  reason  that  edge 
detection  is  so  important  in  both  artificial  and  biological  vision. 

The  coupling  to  intensity  edges  is  done  by  adding  to  Uc  the  term: 


V(l,e)=/3'P(l-ei)Vc(li)  (7) 

with  ej  =  1,0,  depending  on  the  presence  or  absence  of  an  intensity  edge 
between  site  i,J.  This  term  facilitates  formation  of  discontinuities  (that  is, 
//)  at  the  locations  of  intensity  edges. 

These  highly  parallel  algorithms  map  quite  naturally  onto  an  architecture 
such  as  that  of  the  Connection  Machine,  which  consists  of  16K  simple  1-bit 
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processors  with  local  and  global  conr^.*-'on  capabilities.  These  algorithms 
also  map  onto  VLSI  sarchitectures  of  fully  analog  elements  (we  have  success¬ 
fully  experimented  with  a  version  of  Equation  (7)  in  which  I  is  a  continuous 
variable),  mixed  analog  and  digital  components  (such  as  directly  suggested 
by  the  previous  equations),  and  purely  digital  processors  (similar  to  a  much 
simplified  and  specialized  Connection  Machine). 


2.4.  Recognition 


The  problem  of  visual  recognition  is  a  many-faceted  problem.  We  have  worked 
on  seversd  of  these  sub-problems.  In  particular,  we  summarize  here  some  of 
our  initial  work  on  the  critical  issues  of  how  to  segment  an  image  in  order 
to  initiate  the  recognition  process  and  reduce  its  intrinsic  combinatorics,  and 
how  to  use  2-D  views  to  recognize  3-D  objects. 

2.4.1.  Control  Structures  for  Image  Segmentation 

J.  Mahoney  has  begun  to  explore  the  problem  of  image  segmentation  from 
a  new  perspective.  The  separation  of  figure  from  ground  is  still  very  much 
a  mystery,  despite  progress  in  “higher-level”  matters  such  as  interpretation 
of  three-dimensional  structure  and  object  recognition.  Scrutiny  of  traditional 
information  processing  decompositions  of  visuad  objects  suggests  that  the  root 
of  the  difficulty  is  that  segmentation  is  commonly  assumed  to  be  implemented 
by  a  single  module. 

Mahoney  argues  that  segmentation  should  instead  be  viewed  as  inher¬ 
ently  implicit  in  a  number  of  other  processes  -  i.e.,  the  basic  operations  of 
segmentation  are  selected  and  controlled  by  a  number  of  other  processes.  He 
proposed  an  alternative  system  architecture  that  does  away  with  the  notion 
of  segmentation  as  a  module:  figure-groimd  separation  is  seen  as  a  complex 
behavior  exhibited  by  the  system  as  a  whole,  and  implemented  by  the  in¬ 
teraction  between  its  modifies.  The  modules  of  the  system  axe  elementary 
processes  of  visual  object  perception:  structure  interpretation,  classification 
and  application.  The  control  of  segmentation  exerted  by  these  processes  has 
a  layered  organization:  interpretation  determines  how  figures  are  extracted, 
classification  influences  what  figures  are  extracted,  and  application  determines 
where  and  when  the  figure  separation  takes  place.  This  layered  organization 
implies  that  the  various  levels  of  control  in  separation  may  be  studied  some¬ 
what  independently. 


16 


Our  recent  studies  of  the  basic  segmentation  operations  (curve  tracing, 
area  coloring  and  indexing  of  salient  locations)  addressed  the  question  of  how 
fine-grained  parallelism  of  the  sort  provided  by  the  Connection  Machine  may 
be  exploited  in  implementing  these  operations  efficiently.  Our  study  of  the 
proposed  control  structure  for  segmentation  has  begun  to  examine  the  ele¬ 
mentary  object  perception  processes  -  structure  interpretation,  classification 
smd  application  -  and  their  interactions  in  detail.  The  Connection  Machine 
provides  large-scale  parallelism  and  general  purpose  routing  which  supports 
efficient  implementation  of  these  processes. 


2.4.2.  Recognizing  3-D  Objects  from  2-D  Views  A, 

We  have  implemented  a  system  for  recognizing  objects  in  three-space  from 
a  single  two-dimensional  view,  such  as  a  photograph.  D.  Huttenlocher  and 
S.  Ullman  have  shown  that  three  corresponding  points  are  sufficient  to  re¬ 
cover  the  three-dimensional  position,  orientation  and  scale  of  a  model  with 
respect  to  an  image.  Thus  our  approach  is  to  first  align  models  with  an 
image,  using  triples  of  corresponding  model  and  image  points,  and  then  to 
compare  the  aligned  models  with  the  image.  Unlike  conventional  tree  search 
and  graph  matching  approaches  to  recognition,  our  method  is  well  suited  to 
implementation  on  a  massively  parallel  computer.  Each  aligmnent  operation 
is  independent,  EUid  thus  all  the  alignments  between  models  and  an  image  can 
be  solved  in  parallel. 

Given  a  set  of  corresponding  triples  of  model  and  image  points,  each 
triple  is  used  to  compute  the  position,  orientation,  and  scale  of  all  models 
containing  that  triple.  This  operation  is  performed  in  parallel  for  eJI  triples. 
The  aligned  models  are  then  sorted  to  group  together  instances  of  the  same 
model.  This  takes  time  0(k)  for  2*  models.  Finally,  for  each  model,  the 
number  of  similar  alignments  is  determined.  Models  for  which  there  are  two 
or  more  independent  alignments  proposing  a  similar  position,  orientation  emd 
scale  are  likely  to  actually  be  in  the  image,  and  should  be  further  verified. 


2.5.  Other  Vision  Algorithms 


We  have  explored  the  implementation  on  the  Connection  Machine  of  several 
other  algorithms,  especially  those  with  a  more  global  flavor  requiring  extensive 
use  of  the  router  network  as  opposed  to  the  NEWS  network. 
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2.5.1.  The  Hough  Transform 

The  Hough  Transform  is  frequently  used  in  image  analysis  to  determine  the 
existence  of  straight  lines  in  an  image  [Ballard  and  Brown,  1981].  The  Gen¬ 
eralized  Hough  Transform,  similarly,  is  used  to  determine  the  parameters  of 
the  position  and  orientation  of  a  known  object  from  an  image.  The  Hough 
Transform  computes  an  accumulator  array  of  values,  in  which  the  {6,  p)  entry 
records  the  nxxmber  of  pixels  lying  on  a  line  with  parameters  (6,  p). 

We  parameterize  lines  by  the  normal  angle  and  the  perpendicular  dis¬ 
tance  from  the  origin  p.  The  Hough  Transform  table  will  be  stored  in  a  matrix 
of  processors  indexed  by  {B,  p).  We  compute  the  Hough  Transform  in  B  sepa¬ 
rate  stages;  each  step  computes  the  values  for  a  particular  angle  B(i).  For  each 
2uigle,  we  broadcast  co3B(i)  and  sinB(i).  Each  processor  computes  the  scalar 
product  of  its  (r,  y)  address  with  the  normal  described  by  the  broadcast  pair. 
Each  processor  then  knows  its  {B,p).  We  count  votes  by  sending  a  1  from  each 
active  pixel,  summing  at  the  destination  processor.  If  we  were  simply  to  send 
each  vote  to  the  table,  there  would  be  many  collisions,  with  many  messages 
arriving  at  one  processor,  especially  when  {B,p)  does  in  fact  represent  a  line. 
This  can  be  slow  when  the  number  of  collisions  is  large  (>  64). 

The  routing  hardware  incurs  little  penalty  for  having  up  to  32  collisions 
per  destination.  By  randomizing  the  vote  destinations,  we  can  minimize  the 
number  of  collisions.  For  each  B,  a  pixel  computes  a  location  based  on  rho 
and  some  random  value,  say,  one  computed  at  the  start.  For  a  fixed  p,  pixels 
send  their  votes  to  a  contiguous  block  of  processors.  Block  sizes  are  calculated 
to  reduce  the  average  number  of  collisions.  The  votes  are  summed  when  they 
arrive,  using  a  built-in  capability  of  the  router.  Then  the  votes  are  tallied 
as  before,  using  a  plus-scan  operation,  with  segments.  This  will  achieve  the 
same  result  with  less  overhead  for  computing  addresses. 

2.5.2.  Matching  2-D  Descriptions  with  3-D  Rock  Models 

More  work  has  been  done  on  the  matcher  discussed  by  W.  Lim  [in  Proceedings 
of  the  Image  Understanding  Workshop,  1987).  The  matcher  discussed  in  that 
work  uses  2-D  descriptions,  represented  by  a  graph  termed  a  model  graph, 
for  recognizing  landmarks  which  are  objects  in  the  rocks  world.  The  2-D 
descriptions  are  derived  from  contour  segments  of  the  occluding  contour  of 
the  object.  The  matcher  has  been  implemented  on  the  Connection  Machine 
and  has  been  tested  on  objects  using  manually  created  models  for  three  real 
objects.  Sjmthetic  models  are  also  used  by  changing  some  of  the  attributes 
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in  the  models  for  the  real  objects.  The  number  of  models  in  the  data  b2ise 
is  approximately  one-tenth  the  number  of  nodes  in  the  CM,  since  each  model 
graph  has  about  ten  nodes  as  well  as  nodes  mapped  directly  to  processors. 
The  sizes  of  the  CMs  used  are  8K  and  16K.  Due  to  insufficient  memory  in 
the  processors  of  the  CM-1,  implementation  of  the  algorithm  has  been  moved 
to  the  CM-2,  access  to  which  Thinking  Machines  Corporation  has  provided 
access. 

The  sdgorithm  has  been  extended  to  match  graphs  that  axe  more  com¬ 
plete,  i.e.,  they  contain  2-D  descriptions  derived  from  the  internal  structure 
of  the  objects  in  addition  to  those  obtained  from  the  occluding  contour.  The 
additional  information  includes  the  number  of  observed  surface  patches  that 
are  not  on  the  occluding  contour,  and  possibly  their  surface  types,  e.g.,  flat  or 
curved.  The  algorithm  first  finds  views  of  models  that  match  the  occluding 
contom*.  From  this  set  of  views,  it  then  looks  for  those  that  will  edso  satisfy  the 
constraints  imposed  by  the  internal  structure  of  the  object  in  the  scene.  Work 
is  in  progress  on  implementing  the  extension  and  testing  the  entire  algorithm 
using  larger  data  bases,  perhaps  as  large  as  will  fit  on  the  64K  CM-2. 

2.6.  Other  Architectures 

Most  of  our  work  has  been  focused  on  the  Connection  Machine,  as  we  had 
originally  proposed,  in  order  to  establish  the  strength  and  utility  of  fine¬ 
grained  parallel  architectures  for  vision  within  a  navigation  task.  Thanks 
to  the  in-house  work  by  R.  Brooks  on  the  Mobile  Robot,  we  have  found  it 
interesting  to  contrast  the  fine-grmned  architecture  of  the  CM-1  with  the 
subsximption  architecture  used  by  Brooks. 

2.6.1.  Architectures  for  Robot  Navigation 

We  have  examined  system  architectures  for  robotic  navigation,  in  the  context 
of  Brooks’  layered  systems  for  building  robots.  Brooks’  subsumption  archi¬ 
tecture  for  robots  provides  an  environment  for  building  robots  which  allows 
us  to  investigate  several  forms  of  exploratory  behavior.  In  this  system,  we  can 
study  various  forms  of  spatial  reasoning  [Connell,  1986]. 

Recently,  we  have  been  exploring  the  application  of  Brooks’  subsvunption 
architecture  to  robotic  navigation.  This  architecture  is  attractive  because  a 
complete  system  can  be  built  and  debugged  incrementally.  The  architecture 
also  allows  for  a  graceful  degradation  of  performance  if  the  robot  encounters 
unusual  environments,  or  if  some  of  the  subsystems  become  inoperative. 
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To  test  these  ideas,  we  have  investigated  two  types  of  exploratory  behav¬ 
ior  cast  in  the  subsumption  framework  [Brooks  and  Connell,  1986].  We  are 
also  looking  at  how  to  build  and  maintaiin  coarse  maps  of  the  environment 
using  similar  techniques  [Brooks  and  Connell,  1987].  Finally,  we  are  probing 
ways  to  extend  the  architecture  to  allow  certain  other  types  of  spatial  reason¬ 
ing.  In  particular,  we  are  interested  in  having  the  robot  identify,  pick  up,  and 
then  return  certain  types  of  objects  [Connell,  1986]. 

The  conclusion  we  have  reached  is  that  the  two  architectures  can  be  com¬ 
plementary  for  navigation.  The  simple  subsumption  architecture  can  underly 
simple  reflexive  behaviors  of  the  insect  type.  For  more  sophisticated  tasks 
involving  planned  visual  navigation  and  recognition,  however,  the  power  of  a 
parallel  supercomputer  is  barely  sufficient,  given  the  complexity  of  the  tasks. 

2.7.  Parallel  Languages 

*LISP  is  a  good  language  for  early  and  middle  vision.  For  more  abstract 
tasks,  other  languages  would  be  desirable.  It  is  questionable  whether  Thinking 
Machines  Corporation  will  be  able  to  develop  these  more  abstract  languages, 
since  its  main  effort  at  present  is  focused  on  extending  standard  languages  such 
as  C  and  even  FORTRAN.  We  have  been  compairing  three  paradlel  versions 
of  Lisp  that  have  been  designed  for  the  Connection  Machine.  These  three 
languages  are  *LISP,  Paralation  Lisp  and  CM-Lisp.  *LISP  is  the  cturrent 
parallel  Lisp  supported  by  Thinking  Machines  for  the  Connection  Machine. 
CM-Lisp  is  a  language  designed  by  D.  Hillis  and  G.  Steele  for  the  Connection 
Machine:  it  is  a  much  higher  level  language.  A  simulator  and  a  preliminary 
compiler  have  been  implemented  at  Thinking  Machines.  Paralation  Lisp  is  a 
language  designed  by  G.  Sabot,  a  graduate  student  at  Harvard  University.  It 
is  similau:  to  CM-Lisp,  but  is  not  at  such  a  high  level  (its  model  is  slightly  closer 
to  the  mau:hine  model).  A  simulator  for  the  language  has  been  implemented 
by  Sabot. 

The  purpose  of  comparing  the  three  languages  is  to  determine  what  the 
best  ab8trau:tions  aue  for  a  parallel  Lisp,  and  also  to  select  the  features  of  the 
three  languages  that  are  most  important.  We  feel  that  this  knowledge  will 
help  us  to  decide  on  implementation  languages  in  the  future,  and  will  reduce 
our  coding  effort. 

The  general  approach  tadken  is  to  code  the  same  algorithms  in  the  three 
languages  and  to  get  a  feeling  of  what  helpful  features  are  either  present  or 
lacking  in  each  language.  We  also  try  to  get  a  feeling  of  which  language  most 
cleaurly  and  concisely  expresses  the  algorithms.  In  addition,  we  are  studying 
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compilers  for  the  three  languages  so  that  we  can  determine  how  well  a  com¬ 
piler  can  compile  the  algorithms  into  an  efficient  set  of  Connection  Machine 
instructions. 

Some  subgoals  attained  this  summer  include  the  implementation  of  sev¬ 
eral  algorithms  in  Paxalation  Lisp,  for  example: 

A  sorting  algorithm  ( Quicksort ) 

A  convex  hull  algorithm  (Quickhull) 

A  line  drawing  algorithm 

A  K~D  tree  algorithm 

An  algorithm  to  determine  the  Shannon  entropy  of  a  sequence  of  symbols 
An  algorithm  to  build  Quinlan  Decision  trees. 

We  have  also  sped  up  the  simulator  for  Paralation  Lisp  by  a  factor  of  100, 
and  implemented  a  preliminauy  compiler  for  Paralation  Lisp.  In  addition,  we 
have  implemented  some  algorithms  in  CM-Lisp  including: 

A  sorting  algorithm  (Quicksort) 

A  line  drawing  algorithm 

An  algorithm  to  determine  the  Shannon  entropy  of  a  sequence  of  symbols. 
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